Distinguish router and non-router replicas - #4101
Open
jvstme wants to merge 3 commits into
Open
Conversation
**Problem**: Non-router replicas in a service with a router should not be registered on the gateway. To determine whether a particular replica should be registered, one has to parse its job spec and run spec and see if it is a router replica or not. This does not work well for the upcoming gateway state sync mechanism, which needs to quickly fetch all jobs to be registered from the database, without parsing their specs. **Solution**: - Do **not** set `JobModel.registered` for non-router replicas in services with a router. That way, `JobModel.registered` can be reliably used to identify the jobs that should be registered on the gateway, without parsing the job spec. - Since the router worker sync pipeline needs both router and non-router replicas with successful probes, introduce `JobModel.ready`, set for any replica once probes pass. ```shell $ dstack event --within-run prefill-decode | grep -E "ready|register" [2026-08-05 10:30:29] [job prefill-decode-0-0] Service replica ready to receive requests [2026-08-05 10:30:29] [job prefill-decode-0-0] Service replica registered to receive requests [2026-08-05 10:44:45] [job prefill-decode-0-1] Service replica ready to receive requests [2026-08-05 10:45:00] [job prefill-decode-0-2] Service replica ready to receive requests [2026-08-05 11:15:22] [job prefill-decode-0-0] Service replica unregistered from receiving requests ```
…ish_router_and_non_router_replicas
Collaborator
|
IIUC, the rolling deployment logic also needs to be updated to use "ready" instead of "registered" to mean if the replica is ready to receive service requests. See and dstack/src/dstack/_internal/server/services/runs/replicas.py Lines 80 to 82 in c2484d6 suspicious why the rolling deployment tests do not cover it. |
r4victor
reviewed
Aug 10, 2026
Comment on lines
-116
to
-119
| if router_group is not None and job_spec.replica_group != router_group.name: | ||
| # Strict router-only: when a router is configured, the proxy should only be aware | ||
| # of router replicas. | ||
| continue |
Collaborator
There was a problem hiding this comment.
What about Dynamo workers? I suppose registered remains True for dynamo workers, which means you cannot rely on registered=True to filter router jobs.
Collaborator
Author
There was a problem hiding this comment.
registered doesn't depend on the router type, so Dynamo workers also have registered=False and are filtered out here
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: Non-router replicas in a service with
a router should not be registered on the gateway.
To determine whether a particular replica should
be registered, one has to parse its job spec and
run spec and see if it is a router replica or not.
This does not work well for the upcoming gateway
state sync mechanism, which needs to quickly fetch
all jobs to be registered from the database,
without parsing their specs.
Solution:
JobModel.registeredfornon-router replicas in services with a router.
That way,
JobModel.registeredcan be reliablyused to identify the jobs that should be
registered on the gateway, without parsing the
job spec.
router and non-router replicas with successful
probes, introduce
JobModel.ready, set for anyreplica once probes pass.
#3959